Skip to content

feat(review): opt-in test-gap check and focused AI batches; verifier reads every changed file - #1772

Open
RaghwendraSingh wants to merge 7 commits into
feat/review-ai-input-coveragefrom
feat/review-ai-focused-batches
Open

RaghwendraSingh wants to merge 7 commits into
feat/review-ai-input-coveragefrom
feat/review-ai-focused-batches

Conversation

@RaghwendraSingh

@RaghwendraSingh RaghwendraSingh commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Issue for this PR

Part of #1792

Type of change

  • Bug fix
  • New feature
  • Refactor / code improvement
  • Documentation

What does this PR do?

Stacked on #1771. Adds two opt-in AI-lane options, lets the verifier read every changed file, and fixes the rule gaps the planted-bug suite found. Defaults are unchanged.

What changed

  • ALTIMATE_REVIEW_AI_TEST_GAP=1: when a changed macro, materialization or generic test changes runtime behaviour, the AI says whether the PR adds a test that runs it, as one test_coverage comment. Each batch also gets the PR's other changed paths.
  • ALTIMATE_REVIEW_AI_FOCUSED=1: on PRs with 7 or more files, related files are reviewed together in batches of up to 5. Related means a macro with its dispatch variants, a test with the model it refs, and a yml with the models it declares. Grounding is filtered per batch, findings are deduplicated, and there are at most 4 batches. No paid measurement yet, so it stays opt-in.
  • The verifier can read every file the PR changes, so it can confirm "no test was added".
  • Rule fixes from the planted-bug suite:
    • three recall rules;
    • the join chain across a join with no ON clause;
    • parse-breaking findings are kept in dev, sandbox and scratch folders.

Results

Test-gap check

  • A private 5-project monorepo PR that changed a materialization with no test: the AI raised the missing test in 10 of 10 runs, against 2 of 21 before.
  • 10 tuning-repo PRs: catches unchanged (5 of 16), useful comments 51% → 57%, wrong 3% → 0% (blind Claude judge), tokens 1.02×.

Planted-bug suite on repos never used for tuning: 236 of 238 caught, 1 false alarm on 237 no-defect twins.

How did you verify your code works?

  • bun test test/altimate/review: 748 pass.
  • bunx tsc --noEmit: clean.
  • New: test/altimate/review-ai-test-gap.test.ts, plus focused-batch cases in review-ai-inputs.test.ts.

Cost side effects

None by default; both options are opt-in. ALTIMATE_REVIEW_AI_TEST_GAP=1 measured at about 1.02x AI-lane tokens. ALTIMATE_REVIEW_AI_FOCUSED=1 splits PRs with 7 or more files into at most 4 batches, so up to 4 AI calls per review instead of one (not yet measured, hence opt-in). Letting the verifier read every changed file only applies when the verifier is enabled and stays within its 12k-token cap. No warehouse queries or cloud resources.

Screenshots / recordings

Not a UI change.

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR
  • I checked for cost side effects (model calls, queries or cloud resources that grow with user data) and described them, or wrote "none"

🤖 Generated with Claude Code

@coderabbitai

coderabbitai Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Repository UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 921c8c98-36ce-4488-a033-4b8c7f8a8b5f

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

const others = [...new Set([...newContent.matchAll(DOC_REF)].map((m) => m[1]))].filter((n) => !missing.includes(n)).slice(0, 5)
let resolves = false
for (const n of others) if ((resolves = await docExists(n))) break
if (!others.length && projectHasDocs) resolves = await projectHasDocs()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WARNING: projectHasDocs() cannot tell a project docs block from one that resolves in an installed package

docExists only greps the dbt project's own paths (grepProject pathspecs come from the dbt_project.yml model/macro/seed/snapshot/test dirs, run.ts:431-438), so a doc('name') defined by an installed package already counts as "missing". This new fallback then treats any docs block anywhere in the project as proof that the name resolves, so for a file whose only doc() is the new one (typically a newly added schema yml) a package-provided doc is reported as "dbt fails to parse" — exactly the package case the guard above was added to avoid. The finding body also says "the other doc() references in this file resolve there", which is false on this branch (there are none). Consider also searching installed-package .md files before concluding the name is missing, or keep the previous "another doc() in this file resolves" requirement for names that could come from a package.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 31cb5c4: doc() names defined in the .md files of installed packages (dbt_packages/) now count as defined. The no-other-doc() fallback also requires the packages' blocks to be known (no packages declared, or packages installed), so a package-provided doc() stays silent when packages aren't installed. The body now says why the finding is trusted ("the other doc() references resolve" or "the project defines its own docs blocks") and no longer claims the first on the fallback path.

// The project defines docs blocks of its own (any `{% docs x %}` in its .md files).
let docsAny: Promise<boolean> | undefined
const projectHasDocs = () =>
(docsAny ??= (input.grepProject?.("docs", "*.md") ?? Promise.resolve([]))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: the generic docs pattern can exhaust the search cap before reaching a docs block

grepFixedAt stops after 50 total hits (git.ts:194-210; -m 50 per file), and docs shows up in ordinary markdown prose, so those hits can be used up before the file holding {% docs ... %} is scanned. projectHasDocs then returns false and the new doc-block-missing finding stays silent — the recall gap this change targets. A block-specific fixed string (e.g. separate searches for {% docs and {%- docs) would keep unrelated prose from diluting the result.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up: low priority, tracked. Hitting the cap makes the rule go silent (a missed finding, not a wrong one); grepping for {% docs / {%- docs fixes it.

const TEST_GAP_FILES_MAX = 40

/** The PR's other changed paths, appended to a batch message for the test-gap instruction. */
export function testGapFileList(message: string, batch: AiReviewFile[], prFiles: string[] = []): string {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: three copies of the same "other PR paths" renderer

This function, the focused-batch ## OTHER FILES CHANGED IN THIS PR list in buildBatchMessage (lines 777-782), and buildVerifyMessage's "Files this PR changes" list (ai-verify.ts:386-402) each implement slice(0, 40) + "… and N more" with their own 40 constant (TEST_GAP_FILES_MAX, FOCUS_OTHER_FILES, PR_FILES_MAX). With ALTIMATE_REVIEW_AI_FOCUSED=1 and ALTIMATE_REVIEW_AI_TEST_GAP=1, the same batch message carries both lists and repeats every other path in the prompt. A single shared helper (and cap) would remove the duplication and the double listing.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair; we'll fold them into one renderer in a cleanup pass.

* be dropped, or null to keep it. Centralizing this here (vs prompt text) is
* what keeps the false-positive rate down deterministically.
*/
/** Findings that break `dbt parse` for the whole project, so a dev/sandbox/scratch folder doesn't excuse them. */

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: the exclusionReason JSDoc now documents this constant

The /** Apply the rubric's exclusion predicates ... */ block above now attaches to PROJECT_BREAKING_RULES, so exclusionReason loses its doc and editors/TS show the wrong description for it. Move the constant above the JSDoc, or give the constant its own comment.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch on the doc placement; we'll move it in the next touch of rubric.ts. (kilo, misplaced exclusionReason JSDoc)

@kilo-code-bot

kilo-code-bot Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Code Review Summary

Status: 5 Issues Found | Recommendation: Address before merge

Overview

Severity Count
CRITICAL 0
WARNING 2
SUGGESTION 3
Issue Details (click to expand)

WARNING

File Line Issue
packages/opencode/src/altimate/review/rubric.ts 86 The new parse-breaking exemption set omits dbt-project-rules:ref-target-missing (whose body says dbt fails to parse), so a typo'd ref() under dev/, sandbox/ or scratch/ is still dropped with skipNonProdModels on by default
packages/opencode/src/altimate/review/ai-review.ts 583 packFocusedBatches never consults budget.batchChars, so with focused + coverage flags a single call can carry ~220k chars against the 120k per-call budget COVERAGE_BATCH_CHARS exists to enforce

SUGGESTION

File Line Issue
packages/opencode/src/altimate/review/run.ts 620 packageDocNames reads packages.yml from the commit tree but package docs from the working tree; an uncommitted packages.yml yields an authoritative empty set and a false doc-block-missing (mirror projectResourceNames' working-tree merge)
packages/opencode/src/altimate/review/ai-review.ts 894 The test-gap file list is appended after the "Return ONLY the JSON array" line, so the output contract is no longer final and plan.messages no longer matches what is sent
packages/opencode/test/altimate/review-project-rules.test.ts 736 The added single-quote escape case is excluded by the rule's log( unless clause, so it cannot fail and the new (?<!\\) guard is untested
Files Reviewed (17 files)
  • packages/opencode/src/altimate/review/ai-review.ts - 2 issues
  • packages/opencode/src/altimate/review/ai-verify.ts - clean
  • packages/opencode/src/altimate/review/dbt-patterns.ts - clean
  • packages/opencode/src/altimate/review/diff-filter.ts - clean
  • packages/opencode/src/altimate/review/orchestrate.ts - clean
  • packages/opencode/src/altimate/review/project-rules.ts - clean
  • packages/opencode/src/altimate/review/rubric.ts - 1 issue
  • packages/opencode/src/altimate/review/rule-catalog.ts - clean
  • packages/opencode/src/altimate/review/run.ts - 1 issue
  • packages/opencode/src/altimate/review/verdict.ts - clean
  • packages/opencode/test/altimate/fixtures/review-ai-prompt-coverage.txt - clean
  • packages/opencode/test/altimate/review-ai-inputs.test.ts - clean
  • packages/opencode/test/altimate/review-ai-test-gap.test.ts - clean
  • packages/opencode/test/altimate/review-ai-verify.test.ts - clean
  • packages/opencode/test/altimate/review-dbt-patterns.test.ts - clean
  • packages/opencode/test/altimate/review-project-rules.test.ts - 1 issue
  • packages/opencode/test/altimate/review.test.ts - clean

Fix these issues in Kilo Cloud

Previous Review Summary (commit 1acd5d8)

Current summary above is authoritative. Previous snapshots are kept for context only.

Previous review (commit 1acd5d8)

Status: 4 Issues Found | Recommendation: Address before merge

Overview

Severity Count
CRITICAL 0
WARNING 1
SUGGESTION 3
Issue Details (click to expand)

WARNING

File Line Issue
packages/opencode/src/altimate/review/project-rules.ts 506 New projectHasDocs() fallback treats any project docs block as proof the new doc() resolves, so a package-provided doc in a file whose only doc() is the new one is reported as a parse error (and the body text about "other doc() references" is false on this branch)

SUGGESTION

File Line Issue
packages/opencode/src/altimate/review/orchestrate.ts 2287 Generic docs fixed-string search can consume the 50-hit grepFixedAt cap on markdown prose before reaching the docs block, silently returning false
packages/opencode/src/altimate/review/ai-review.ts 113 Three copies of the "other PR paths" list renderer (test-gap, focused batches, verifier) with three separate 40 caps; both flags on duplicates every other path in a batch message
packages/opencode/src/altimate/review/rubric.ts 78 The exclusionReason JSDoc now attaches to the new PROJECT_BREAKING_RULES constant, leaving the function undocumented/misdocumented
Files Reviewed (16 files)
  • packages/opencode/src/altimate/review/ai-review.ts - 1 issue
  • packages/opencode/src/altimate/review/ai-verify.ts - clean
  • packages/opencode/src/altimate/review/dbt-patterns.ts - clean
  • packages/opencode/src/altimate/review/orchestrate.ts - 1 issue
  • packages/opencode/src/altimate/review/project-rules.ts - 1 issue
  • packages/opencode/src/altimate/review/rubric.ts - 1 issue
  • packages/opencode/src/altimate/review/rule-catalog.ts - clean
  • packages/opencode/src/altimate/review/run.ts - clean
  • packages/opencode/src/altimate/review/verdict.ts - clean
  • packages/opencode/test/altimate/fixtures/review-ai-prompt-coverage.txt - clean
  • packages/opencode/test/altimate/review-ai-inputs.test.ts - clean
  • packages/opencode/test/altimate/review-ai-test-gap.test.ts - clean
  • packages/opencode/test/altimate/review-ai-verify.test.ts - clean
  • packages/opencode/test/altimate/review-dbt-patterns.test.ts - clean
  • packages/opencode/test/altimate/review-project-rules.test.ts - clean
  • packages/opencode/test/altimate/review.test.ts - clean

Fix these issues in Kilo Cloud


Reviewed by deepseek-v4.1-flash-1017 · Input: 223K · Output: 38.8K · Cached: 4.3M

Review guidance: REVIEW.md from base branch feat/review-ai-input-coverage

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

10 issues found across 16 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/opencode/src/altimate/review/run.ts">

<violation number="1" location="packages/opencode/src/altimate/review/run.ts:440">
P2: `prFiles` includes deleted files, so a PR that removes a test file still shows that test under "PR's other changed paths" and the test-gap check may treat it as coverage. Filter out `status === "deleted"` before mapping to paths.</violation>
</file>

<file name="packages/opencode/test/altimate/review-ai-test-gap.test.ts">

<violation number="1" location="packages/opencode/test/altimate/review-ai-test-gap.test.ts:29">
P3: This test depends on the real native `altimate_core.review_ai_prompt` call, but it never stubs `Dispatcher.call`. If the native core is unavailable, `runAiReview` returns early and `LLM.stream` is never called, so the test fails on an undefined `off`. Stub it with `spyOn(Dispatcher, "call").mockResolvedValue({ success: true, data: { prompt: "CORE" } } as any)`, as the replay test does.</violation>
</file>

<file name="packages/opencode/src/altimate/review/ai-review.ts">

<violation number="1" location="packages/opencode/src/altimate/review/ai-review.ts:117">
P2: For PRs with more than 40 other paths, later test-file names disappear from the reviewer’s prompt, so an unselected test can be missed and trigger a false test-gap comment. Preserve test-like paths beyond the cap or let the reviewer enumerate all PR paths.</violation>

<violation number="2" location="packages/opencode/src/altimate/review/ai-review.ts:579">
P2: With `ALTIMATE_REVIEW_AI_MAX_BATCHES` between 0 and 1 (e.g. 0.5), `Array.from({ length: maxBatches })` creates no bins and the `reduce` throws, and `runAiReview` rejects instead of returning a status. Clamp the bin count to a positive integer.</violation>
</file>

<file name="packages/opencode/src/altimate/review/dbt-patterns.ts">

<violation number="1" location="packages/opencode/src/altimate/review/dbt-patterns.ts:933">
P2: A `join ... using (...)` has no ON clause, but this change does not step over it, so the join chain still breaks there and a redundant predicate on the next join is missed. Let the bare-join match also consume a `USING (...)` list.</violation>
</file>

<file name="packages/opencode/src/altimate/review/ai-verify.ts">

<violation number="1" location="packages/opencode/src/altimate/review/ai-verify.ts:395">
P2: The verifier cannot discover changed files after the first 40 paths, so a test added at path 41+ can be missed by the test-gap check; add a bounded file-list tool or otherwise expose all paths to the verifier.</violation>

<violation number="2" location="packages/opencode/src/altimate/review/ai-verify.ts:783">
P2: The verifier's read allowlist now includes every PR file, including paths the user excluded with `exclude` or rubric `excludeGlobs` and out-of-project files. `prFiles` comes from the unfiltered `collected` list in run.ts, so those contents can reach the model provider. Build `prFiles` from the reviewable set after exclusions, or filter the allowlist with the same exclusion rules.</violation>
</file>

<file name="packages/opencode/src/altimate/review/orchestrate.ts">

<violation number="1" location="packages/opencode/src/altimate/review/orchestrate.ts:2287">
P2: The generic `docs` search can exhaust `grepFixedAt`’s 50-hit limit on Markdown prose before reaching a docs block, causing `projectHasDocs()` to suppress missing-`doc()` findings. Use a targeted, complete docs-block search instead.</violation>
</file>

<file name="packages/opencode/src/altimate/review/rule-catalog.ts">

<violation number="1" location="packages/opencode/src/altimate/review/rule-catalog.ts:775">
P3: The lookbehind exempts any seam character preceded by a backslash, so a literal ending in `\\n` (escaped backslash plus `n`) also escapes detection even though rendered SQL still ends in a word character. Check backslash parity before suppressing the match.</violation>
</file>

<file name="packages/opencode/src/altimate/review/rubric.ts">

<violation number="1" location="packages/opencode/src/altimate/review/rubric.ts:79">
P3: Move `PROJECT_BREAKING_RULES` above the `exclusionReason` JSDoc or give the constant its own comment so the predicate documentation remains attached to the function.</violation>
</file>

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

const specs = glob ? [`${root}${glob}`] : dirs.map((d) => path.posix.normalize(root + d))
return grepFixedAt(gitRoot, opts.head, pattern, specs.filter((d) => !d.startsWith("..")))
},
prFiles: collected.map((f) => f.path),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: prFiles includes deleted files, so a PR that removes a test file still shows that test under "PR's other changed paths" and the test-gap check may treat it as coverage. Filter out status === "deleted" before mapping to paths.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/run.ts, line 440:

<comment>`prFiles` includes deleted files, so a PR that removes a test file still shows that test under "PR's other changed paths" and the test-gap check may treat it as coverage. Filter out `status === "deleted"` before mapping to paths.</comment>

<file context>
@@ -437,6 +437,7 @@ export async function reviewPullRequest(opts: ReviewPullRequestOptions): Promise
       const specs = glob ? [`${root}${glob}`] : dirs.map((d) => path.posix.normalize(root + d))
       return grepFixedAt(gitRoot, opts.head, pattern, specs.filter((d) => !d.startsWith("..")))
     },
+    prFiles: collected.map((f) => f.path),
     prTitle: opts.prTitle,
     prBody: opts.prBody,
</file context>
Suggested change
prFiles: collected.map((f) => f.path),
prFiles: collected.filter((f) => f.status !== "deleted").map((f) => f.path),

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up: low priority, tracked. Filtering out deleted would also stop the verifier reading a deleted file's base side, so it needs a small design call first.

}
if (batches.length <= maxBatches) return batches
const size = (c: AiReviewFile[]) => c.reduce((n, f) => n + fileSection(f, budget.maxFileChars, budget.coverage).length, 0)
const bins = Array.from({ length: maxBatches }, () => ({ chars: 0, chunks: [] as number[] }))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: With ALTIMATE_REVIEW_AI_MAX_BATCHES between 0 and 1 (e.g. 0.5), Array.from({ length: maxBatches }) creates no bins and the reduce throws, and runAiReview rejects instead of returning a status. Clamp the bin count to a positive integer.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/ai-review.ts, line 579:

<comment>With `ALTIMATE_REVIEW_AI_MAX_BATCHES` between 0 and 1 (e.g. 0.5), `Array.from({ length: maxBatches })` creates no bins and the `reduce` throws, and `runAiReview` rejects instead of returning a status. Clamp the bin count to a positive integer.</comment>

<file context>
@@ -431,6 +479,126 @@ export function batchAiFiles(files: AiReviewFile[], budget: AiInputBudget): AiRe
+  }
+  if (batches.length <= maxBatches) return batches
+  const size = (c: AiReviewFile[]) => c.reduce((n, f) => n + fileSection(f, budget.maxFileChars, budget.coverage).length, 0)
+  const bins = Array.from({ length: maxBatches }, () => ({ chars: 0, chunks: [] as number[] }))
+  chunks
+    .map((c, i) => ({ i, chars: size(c) }))
</file context>
Suggested change
const bins = Array.from({ length: maxBatches }, () => ({ chars: 0, chunks: [] as number[] }))
const bins = Array.from({ length: Math.max(1, Math.floor(maxBatches)) }, () => ({ chars: 0, chunks: [] as number[] }))

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up: low priority, tracked. The caller catches it and the lane reports error; Math.max(1, Math.floor(...)) fixes it.

if (!tail) {
joins.push({ preds: [], start: m.index!, end: i, inner })
// A join with no ON (`cross join params as p`): step over its alias so the chain continues.
const bare = /^\s+(?:as\s+)?(?!(?:natural|inner|left|right|full|cross|outer|join|lateral|on|using|where|group|order|qualify|having|union|limit|window|select|from)\b)\w+\b/i.exec(s.slice(i))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: A join ... using (...) has no ON clause, but this change does not step over it, so the join chain still breaks there and a redundant predicate on the next join is missed. Let the bare-join match also consume a USING (...) list.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/dbt-patterns.ts, line 933:

<comment>A `join ... using (...)` has no ON clause, but this change does not step over it, so the join chain still breaks there and a redundant predicate on the next join is missed. Let the bare-join match also consume a `USING (...)` list.</comment>

<file context>
@@ -929,7 +929,9 @@ export function redundantJoinPredicates(sql: string): string[] {
     if (!tail) {
-      joins.push({ preds: [], start: m.index!, end: i, inner })
+      // A join with no ON (`cross join params as p`): step over its alias so the chain continues.
+      const bare = /^\s+(?:as\s+)?(?!(?:natural|inner|left|right|full|cross|outer|join|lateral|on|using|where|group|order|qualify|having|union|limit|window|select|from)\b)\w+\b/i.exec(s.slice(i))
+      joins.push({ preds: [], start: m.index!, end: i + (bare?.[0].length ?? 0), inner })
       continue
</file context>
Suggested change
const bare = /^\s+(?:as\s+)?(?!(?:natural|inner|left|right|full|cross|outer|join|lateral|on|using|where|group|order|qualify|having|union|limit|window|select|from)\b)\w+\b/i.exec(s.slice(i))
const bare = /^\s+(?:as\s+)?(?!(?:natural|inner|left|right|full|cross|outer|join|lateral|on|using|where|group|order|qualify|having|union|limit|window|select|from)\b)\w+\b|^\s+using\s*\([^)]*\)/i.exec(s.slice(i))

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up: low priority, tracked. The result is a missed redundant predicate, never a false one.

...ev.addedLines.keys(),
...graph.parents.map((p) => p.path),
...graph.downstream.map((d) => d.path),
...(ev.prFiles ?? []),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: The verifier's read allowlist now includes every PR file, including paths the user excluded with exclude or rubric excludeGlobs and out-of-project files. prFiles comes from the unfiltered collected list in run.ts, so those contents can reach the model provider. Build prFiles from the reviewable set after exclusions, or filter the allowlist with the same exclusion rules.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/ai-verify.ts, line 783:

<comment>The verifier's read allowlist now includes every PR file, including paths the user excluded with `exclude` or rubric `excludeGlobs` and out-of-project files. `prFiles` comes from the unfiltered `collected` list in run.ts, so those contents can reach the model provider. Build `prFiles` from the reviewable set after exclusions, or filter the allowlist with the same exclusion rules.</comment>

<file context>
@@ -763,6 +780,7 @@ export function verifierTools(
       ...ev.addedLines.keys(),
       ...graph.parents.map((p) => p.path),
       ...graph.downstream.map((d) => d.path),
+      ...(ev.prFiles ?? []),
     ]),
   ]
</file context>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 31cb5c4: prFiles (the verifier's read allowlist and the test-gap list) now drops paths that match the user's exclude globs, plus build output and vendored paths (isExcludedPath). Changed files outside the dbt project stay readable on purpose (a Python test elsewhere in the repo is evidence for a test-gap claim), as OrchestrateInput.prFiles documents.

const inBatch = new Set(batch.map((f) => f.path))
const others = prFiles.filter((p) => !inBatch.has(p))
if (!others.length) return message
const shown = others.slice(0, TEST_GAP_FILES_MAX)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: For PRs with more than 40 other paths, later test-file names disappear from the reviewer’s prompt, so an unselected test can be missed and trigger a false test-gap comment. Preserve test-like paths beyond the cap or let the reviewer enumerate all PR paths.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/ai-review.ts, line 117:

<comment>For PRs with more than 40 other paths, later test-file names disappear from the reviewer’s prompt, so an unselected test can be missed and trigger a false test-gap comment. Preserve test-like paths beyond the cap or let the reviewer enumerate all PR paths.</comment>

<file context>
@@ -76,6 +96,34 @@ export function aiDryRunEnabled(env: Record<string, string | undefined> = proces
+  const inBatch = new Set(batch.map((f) => f.path))
+  const others = prFiles.filter((p) => !inBatch.has(p))
+  if (!others.length) return message
+  const shown = others.slice(0, TEST_GAP_FILES_MAX)
+  return [
+    message,
</file context>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up: low priority, tracked. We'll list test-like paths first.

const parts = [`PR: ${(prTitle ?? "(untitled)").slice(0, 200)}`]
// A test-coverage claim is checked against the PR's own test files, which may sit outside the dbt project.
if (prFiles.length && candidates.some((c) => c.finding.category === "test_coverage")) {
const shown = prFiles.slice(0, PR_FILES_MAX)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: The verifier cannot discover changed files after the first 40 paths, so a test added at path 41+ can be missed by the test-gap check; add a bounded file-list tool or otherwise expose all paths to the verifier.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/ai-verify.ts, line 395:

<comment>The verifier cannot discover changed files after the first 40 paths, so a test added at path 41+ can be missed by the test-gap check; add a bounded file-list tool or otherwise expose all paths to the verifier.</comment>

<file context>
@@ -378,12 +380,27 @@ export function hunkAround(diff: string | undefined, line?: number): string {
+  const parts = [`PR: ${(prTitle ?? "(untitled)").slice(0, 200)}`]
+  // A test-coverage claim is checked against the PR's own test files, which may sit outside the dbt project.
+  if (prFiles.length && candidates.some((c) => c.finding.category === "test_coverage")) {
+    const shown = prFiles.slice(0, PR_FILES_MAX)
+    parts.push(
+      "",
</file context>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as r4627044412; tracked.

const seen: any[] = []
spyOn(Provider, "defaultModel").mockResolvedValue({ providerID: "fake", modelID: "m" } as any)
spyOn(Provider, "getModel").mockResolvedValue(FAKE_MODEL as any)
spyOn(LLM, "stream").mockImplementation(async (input: any) => {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: This test depends on the real native altimate_core.review_ai_prompt call, but it never stubs Dispatcher.call. If the native core is unavailable, runAiReview returns early and LLM.stream is never called, so the test fails on an undefined off. Stub it with spyOn(Dispatcher, "call").mockResolvedValue({ success: true, data: { prompt: "CORE" } } as any), as the replay test does.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/test/altimate/review-ai-test-gap.test.ts, line 29:

<comment>This test depends on the real native `altimate_core.review_ai_prompt` call, but it never stubs `Dispatcher.call`. If the native core is unavailable, `runAiReview` returns early and `LLM.stream` is never called, so the test fails on an undefined `off`. Stub it with `spyOn(Dispatcher, "call").mockResolvedValue({ success: true, data: { prompt: "CORE" } } as any)`, as the replay test does.</comment>

<file context>
@@ -0,0 +1,58 @@
+    const seen: any[] = []
+    spyOn(Provider, "defaultModel").mockResolvedValue({ providerID: "fake", modelID: "m" } as any)
+    spyOn(Provider, "getModel").mockResolvedValue(FAKE_MODEL as any)
+    spyOn(LLM, "stream").mockImplementation(async (input: any) => {
+      seen.push(input)
+      return {
</file context>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up: low priority, tracked. We'll stub it so the test doesn't depend on the native prompt.

// Single- or double-quoted.
added:
// An escape (`\n`) at the seam is not a word.
/'[^'\n]*\s[^'\n]*(?<!\\)[A-Za-z0-9]'\s*~\s*[A-Za-z_][\w.]*|[\w)]\s*~\s*'[A-Za-z0-9][^'\n]*\s[^'\n]*'|"[^"\n]*\s[^"\n]*(?<!\\)[A-Za-z0-9]"\s*~\s*[A-Za-z_][\w.]*|[\w)]\s*~\s*"[A-Za-z0-9][^"\n]*\s[^"\n]*"/,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: The lookbehind exempts any seam character preceded by a backslash, so a literal ending in \\n (escaped backslash plus n) also escapes detection even though rendered SQL still ends in a word character. Check backslash parity before suppressing the match.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/rule-catalog.ts, line 775:

<comment>The lookbehind exempts any seam character preceded by a backslash, so a literal ending in `\\n` (escaped backslash plus `n`) also escapes detection even though rendered SQL still ends in a word character. Check backslash parity before suppressing the match.</comment>

<file context>
@@ -769,7 +769,10 @@ const CORRECTNESS: Rule[] = [
+      // Single- or double-quoted.
+      added:
+        // An escape (`\n`) at the seam is not a word.
+        /'[^'\n]*\s[^'\n]*(?<!\\)[A-Za-z0-9]'\s*~\s*[A-Za-z_][\w.]*|[\w)]\s*~\s*'[A-Za-z0-9][^'\n]*\s[^'\n]*'|"[^"\n]*\s[^"\n]*(?<!\\)[A-Za-z0-9]"\s*~\s*[A-Za-z_][\w.]*|[\w)]\s*~\s*"[A-Za-z0-9][^"\n]*\s[^"\n]*"/,
       macros: true,
     },
</file context>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not changing now: an escaped quote inside a Jinja string literal next to ~ is very rare in dbt code, and the cost is a missed suggestion.

* what keeps the false-positive rate down deterministically.
*/
/** Findings that break `dbt parse` for the whole project, so a dev/sandbox/scratch folder doesn't excuse them. */
const PROJECT_BREAKING_RULES = new Set(["dbt-project-rules:yaml-parse-error", "dbt-project-rules:doc-block-missing"])

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: Move PROJECT_BREAKING_RULES above the exclusionReason JSDoc or give the constant its own comment so the predicate documentation remains attached to the function.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/rubric.ts, line 79:

<comment>Move `PROJECT_BREAKING_RULES` above the `exclusionReason` JSDoc or give the constant its own comment so the predicate documentation remains attached to the function.</comment>

<file context>
@@ -75,12 +75,15 @@ export function blockingCategories(rubric: Rubric): Set<string> {
  * what keeps the false-positive rate down deterministically.
  */
+/** Findings that break `dbt parse` for the whole project, so a dev/sandbox/scratch folder doesn't excuse them. */
+const PROJECT_BREAKING_RULES = new Set(["dbt-project-rules:yaml-parse-error", "dbt-project-rules:doc-block-missing"])
+
 export function exclusionReason(finding: Finding, rubric: Rubric): string | null {
</file context>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch on the doc placement; we'll move it in the next touch of rubric.ts. (cubic, same)

@github-actions

Copy link
Copy Markdown

Thanks for updating your PR! It now meets our contributing guidelines. 👍

const isDev = /(^|\/)(dev|sandbox|scratch)\//.test(finding.file)

if (ex.skipNonProdModels && isDev) return "non-prod model (dev/sandbox/scratch)"
if (ex.skipNonProdModels && isDev && !PROJECT_BREAKING_RULES.has(finding.rule ?? "")) return "non-prod model (dev/sandbox/scratch)"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WARNING: ref-target-missing is still dropped in dev/sandbox/scratch folders even though it breaks dbt parse

This exemption only keeps yaml-parse-error and doc-block-missing, but missingRefFindings emits dbt-project-rules:ref-target-missing, whose own body says "so dbt fails to parse the project" (project-rules.ts:828). skipNonProdModels defaults to true, so a ref('typo') added under models/dev/, sandbox/ or scratch/ is still filtered out here — the exact case the new constant is meant to stop. Add dbt-project-rules:ref-target-missing to the set (and jinja-unbalanced, which breaks compilation of every dependent model, if that is in scope for this fix).

else batches.push([...c])
}
if (batches.length <= maxBatches) return batches
const size = (c: AiReviewFile[]) => c.reduce((n, f) => n + fileSection(f, budget.maxFileChars, budget.coverage).length, 0)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WARNING: focused batching ignores the per-call batchChars budget

packFocusedBatches only uses sizes to balance chunks; it never splits a batch when the sections exceed budget.batchChars. batchAiFiles enforces that cap at line 475, and its default under ALTIMATE_REVIEW_AI_COVERAGE=1 is COVERAGE_BATCH_CHARS (120k) precisely so one call does not overrun the context window. With both flags on, whole chunks are packed into bins, so a bin can hold two 5-file chunks (~220k chars at the default ~22k/file), and a smaller ALTIMATE_REVIEW_AI_MAX_BATCHES makes it worse. Split bin contents against budget.batchChars the way batchAiFiles does, or otherwise clamp each call's section size.

if (!files) return undefined
const prefix = projectRootRel ? projectRootRel + "/" : ""
const names = new Set<string>()
if (!files.includes(prefix + "packages.yml") && !files.includes(prefix + "dependencies.yml")) return names

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: a working-tree review treats an uncommitted packages.yml as "no packages"

listFilesAt(cwd, head) falls back to git ls-tree HEAD when head is undefined, so a newly added/uncommitted packages.yml is not in files and this returns an empty set. Callers treat an empty set as authoritative (projectHasDocs then assumes the packages' blocks are known), while dbt_packages below is read from the working tree, so a doc() supplied by an installed package is reported missing. Mirror projectResourceNames, which merges listWorkingTreeFiles when head === undefined (run.ts:655), or return undefined here instead of an empty set.

for (let i = 0; i < batches.length; i += BATCH_CONCURRENCY) {
const slice = batches.slice(i, i + BATCH_CONCURRENCY)
results.push(...(await Promise.all(slice.map((b, j) => reviewBatch(plan.messages[i + j], b, system, model)))))
results.push(...(await Promise.all(slice.map((b, j) => reviewBatch(testGap ? testGapFileList(plan.messages[i + j], b, input.prFiles) : plan.messages[i + j], b, system, model)))))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: appending the test-gap file list at send time breaks the planned message

buildBatchMessage ends with "Return ONLY the JSON array per the output contract.", so the appended ## OTHER FILES THIS PR CHANGES section becomes the last thing the model reads and the contract is no longer final. It also means plan.messages — documented as the messages runAiReview sends exactly (line 818) and what the dry run reports — differs from what is actually sent when ALTIMATE_REVIEW_AI_TEST_GAP=1. Build the list inside buildBatchMessage (before the contract line) so planning and sending stay identical.

expect(seam('{% set q = "select " ~ cols ~ "from " ~ rel %}')).toBe(true)
expect(seam('{% set q = "select " ~ cols ~ " from " ~ rel %}')).toBe(false)
expect(seam('{{ elementary.debug_log("column_snapshot_query - \\n" ~ column_snapshot_query) }}')).toBe(false)
expect(seam("{{ log('query: \\n' ~ q) }}")).toBe(false)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: this case cannot fail, so the new single-quote escape guard is untested

fires() applies the rule's unless clause (rule-catalog.ts:799), and log( matches it, so the assertion is false whether or not (?<!\\) exists in the single-quoted alternative. Use a non-log/print context (a {% set %} or the dbt.datediff(...) call used above) to exercise the new lookbehind; line 749 already covers the log() exclusion.

RaghwendraSingh and others added 7 commits November 20, 2026 13:23
`ALTIMATE_REVIEW_AI_FOCUSED=1` reviews PRs with at least 7 selected files
in several smaller calls of related files instead of one large call.

- `groupAiFiles`: connected components over macro dispatch families
  (`default__x`, `snowflake__x`, `adapter.dispatch('x'`), `ref()` edges and
  yml `- name:` declarations, with a same-directory fallback for files that
  have no edge; components and members keep selection order
- `packFocusedBatches`: chunks of at most 5 files, never split across calls;
  above 4 calls the chunks are balanced by prompt size into exactly 4
- each focused call carries only its own files' grounding and lists the other
  changed files it must not comment on; `ai.input[].batch` records the call
- findings from all calls are deduplicated by file, line and normalized title
  before the verifier
- knobs: `ALTIMATE_REVIEW_AI_FOCUS_MIN_FILES`, `ALTIMATE_REVIEW_AI_FOCUS_BATCH_FILES`,
  `ALTIMATE_REVIEW_AI_MAX_BATCHES`; with the switch unset batches and messages
  are unchanged (coverage-only messages pinned by a new snapshot)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ules

- `doc-block-missing`: a yml whose only `doc()` is the new one now fires
  when the project keeps docs blocks of its own (before, it needed another
  `doc()` in the same file to resolve)
- `yaml-parse-error`: anchor on the added line at or before the parser's
  error line, or the first added line, when the error line is missing or
  past the edit (an unterminated quote)
- `jinja-concat-missing-space`: match double-quoted strings too, and
  ignore an escape such as `\n` at the seam

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`redundant-join-predicate` reset the chain at `cross join params as p`
because the alias was left in the gap before the next join. Step over the
alias of a join that has no ON clause so predicates of earlier inner joins
are still tracked.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… folders

`skipNonProdModels` dropped every finding under `dev/`, `sandbox/` or
`scratch/`. A yml that no longer parses or a `doc()` with no docs block
breaks `dbt parse` for the whole project wherever the file lives, so
`yaml-parse-error` and `doc-block-missing` are exempt.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A test-coverage claim on a dbt file was discarded as unverifiable when the
PR's tests lived outside the dbt project: the verifier could only read the
reviewed dbt files and their graph neighbours.

- `AiVerifyEvidence.prFiles`: every changed path in the PR, including tests
  and CI outside the dbt project; `read_file` may open them
- the verifier message lists those files when a candidate is a
  `test_coverage` claim (other batches are unchanged)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`ALTIMATE_REVIEW_AI_TEST_GAP=1` appends one instruction to the core
reviewer prompt: for a changed macro, materialization or generic test whose
runtime behaviour changes, say whether the PR adds a test that runs it, and
report one `test_coverage` comment when none does. Each batch message also
lists the PR's other changed paths (tests and CI outside the dbt project
included), so the model can tell whether a test exists.

Unset, the system prompt and messages are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…cs blocks

- `prFiles` (the AI test-gap list and the verifier's read allowlist) now drops
  paths matching the user's exclude globs and build output / vendored paths
  (new `isExcludedPath` in `diff-filter.ts`); out-of-project PR files stay readable
  by design.
- `doc-block-missing`: `doc()` names defined in installed packages' `.md` files
  (`dbt_packages/`, new `packageDocNames` in `run.ts`) count as defined. The
  no-other-`doc()` fallback now also requires the packages' docs blocks to be
  known (no packages declared, or installed), so a package-provided `doc()` is not
  reported as a parse failure. The finding body no longer claims other `doc()`
  references resolve when that path did not check them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@RaghwendraSingh
RaghwendraSingh force-pushed the feat/review-ai-input-coverage branch from bf6e861 to c99cc33 Compare November 20, 2026 08:22
@RaghwendraSingh
RaghwendraSingh force-pushed the feat/review-ai-focused-batches branch from 31cb5c4 to ddc4333 Compare November 20, 2026 08:22

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant